{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Processing Anime Data \n",
    "\n",
    "## Background\n",
    "\n",
    "We will use pyjanitor to showcase how to conveniently chain methods together to perform data cleaning in one shot. We \n",
    "We first define and register a series of dataframe methods with pandas_flavor. Then we chain the dataframe methods together with pyjanitor methods to complete the data cleaning process. The below example shows a one-shot script followed by a step-by-step detail of each part of the method chain.\n",
    "\n",
    "We have adapted a [TidyTuesday analysis](https://github.com/rfordatascience/tidytuesday/blob/master/data/2019/2019-04-23/readme.md) that was originally performed in R. The original text from TidyTuesday will be shown in blockquotes.\n",
    "\n",
    "Note: TidyTuesday is based on the principles discussed and made popular by Hadley Wickham in his paper [Tidy Data](https://www.jstatsoft.org/index.php/jss/article/view/v059i10/v59i10.pdf).\n",
    "\n",
    "*The original text from TidyTuesday will be shown in blockquotes.*\n",
    "Here is a description of the Anime data set that we will use."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">This week's data comes from [Tam Nguyen](https://github.com/tamdrashtri) and [MyAnimeList.net via Kaggle](https://www.kaggle.com/aludosan/myanimelist-anime-dataset-as-20190204). [According to Wikipedia](https://en.wikipedia.org/wiki/MyAnimeList) - \"MyAnimeList, often abbreviated as MAL, is an anime and manga social networking and social cataloging application website. The site provides its users with a list-like system to organize and score anime and manga. It facilitates finding users who share similar tastes and provides a large database on anime and manga. The site claims to have 4.4 million anime and 775,000 manga entries. In 2015, the site received 120 million visitors a month.\"\n",
    ">\n",
    ">Anime without rankings or popularity scores were excluded. Producers, genre, and studio were converted from lists to tidy observations, so there will be repetitions of shows with multiple producers, genres, etc. The raw data is also uploaded.\n",
    ">\n",
    ">Lots of interesting ways to explore the data this week!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import libraries and load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import pyjanitor and pandas\n",
    "import pandas as pd\n",
    "import pandas_flavor as pf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Suppress user warnings when we try overwriting our custom pandas flavor functions\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## One-Shot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = \"https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2019/2019-04-23/raw_anime.csv\"\n",
    "df = pd.read_csv(filename)\n",
    "\n",
    "\n",
    "@pf.register_dataframe_method\n",
    "def str_remove(df, column_name: str, pat: str, *args, **kwargs):\n",
    "    \"\"\"Wrapper around df.str.replace\"\"\"\n",
    "\n",
    "    df[column_name] = df[column_name].str.replace(pat, \"\", *args, **kwargs)\n",
    "    return df\n",
    "\n",
    "\n",
    "@pf.register_dataframe_method\n",
    "def str_trim(df, column_name: str, *args, **kwargs):\n",
    "    \"\"\"Wrapper around df.str.strip\"\"\"\n",
    "\n",
    "    df[column_name] = df[column_name].str.strip(*args, **kwargs)\n",
    "    return df\n",
    "\n",
    "\n",
    "@pf.register_dataframe_method\n",
    "def explode(df: pd.DataFrame, column_name: str, sep: str):\n",
    "    \"\"\"\n",
    "    For rows with a list of values, this function will create new\n",
    "    rows for each value in the list\n",
    "    \"\"\"\n",
    "\n",
    "    df[\"id\"] = df.index\n",
    "    wdf = (\n",
    "        pd.DataFrame(df[column_name].str.split(sep).fillna(\"\").tolist())\n",
    "        .stack()\n",
    "        .reset_index()\n",
    "    )\n",
    "    # exploded_column = column_name\n",
    "    wdf.columns = [\"id\", \"depth\", column_name]  # plural form to singular form\n",
    "    # wdf[column_name] = wdf[column_name].apply(lambda x: x.strip())  # trim\n",
    "    wdf.drop(\"depth\", axis=1, inplace=True)\n",
    "\n",
    "    return pd.merge(df, wdf, on=\"id\", suffixes=(\"_drop\", \"\")).drop(\n",
    "        columns=[\"id\", column_name + \"_drop\"]\n",
    "    )\n",
    "\n",
    "\n",
    "@pf.register_dataframe_method\n",
    "def str_word(\n",
    "    df,\n",
    "    column_name: str,\n",
    "    start: int = None,\n",
    "    stop: int = None,\n",
    "    pat: str = \" \",\n",
    "    *args,\n",
    "    **kwargs,\n",
    "):\n",
    "    \"\"\"\n",
    "    Wrapper around `df.str.split`\n",
    "    with additional `start` and `end` arguments\n",
    "    to select a slice of the list of words.\n",
    "    \"\"\"\n",
    "\n",
    "    df[column_name] = df[column_name].str.split(pat).str[start:stop]\n",
    "    return df\n",
    "\n",
    "\n",
    "@pf.register_dataframe_method\n",
    "def str_join(df, column_name: str, sep: str, *args, **kwargs):\n",
    "    \"\"\"\n",
    "    Wrapper around `df.str.join`\n",
    "    Joins items in a list.\n",
    "    \"\"\"\n",
    "\n",
    "    df[column_name] = df[column_name].str.join(sep)\n",
    "    return df\n",
    "\n",
    "\n",
    "@pf.register_dataframe_method\n",
    "def str_slice(\n",
    "    df, column_name: str, start: int = None, stop: int = None, *args, **kwargs\n",
    "):\n",
    "    \"\"\"\n",
    "    Wrapper around `df.str.slice\n",
    "    \"\"\"\n",
    "\n",
    "    df[column_name] = df[column_name].str[start:stop]\n",
    "    return df\n",
    "\n",
    "\n",
    "clean_df = (\n",
    "    df.str_remove(column_name=\"producers\", pat=\"\\[|\\]\")\n",
    "    .explode(column_name=\"producers\", sep=\",\")\n",
    "    .str_remove(column_name=\"producers\", pat=\"'\")\n",
    "    .str_trim(\"producers\")\n",
    "    .str_remove(column_name=\"genre\", pat=\"\\[|\\]\")\n",
    "    .explode(column_name=\"genre\", sep=\",\")\n",
    "    .str_remove(column_name=\"genre\", pat=\"'\")\n",
    "    .str_trim(column_name=\"genre\")\n",
    "    .str_remove(column_name=\"studio\", pat=\"\\[|\\]\")\n",
    "    .explode(column_name=\"studio\", sep=\",\")\n",
    "    .str_remove(column_name=\"studio\", pat=\"'\")\n",
    "    .str_trim(column_name=\"studio\")\n",
    "    .str_remove(column_name=\"aired\", pat=\"\\{|\\}|'from':\\s*|'to':\\s*\")\n",
    "    .str_word(column_name=\"aired\", start=0, stop=2, pat=\",\")\n",
    "    .str_join(column_name=\"aired\", sep=\",\")\n",
    "    .deconcatenate_column(\n",
    "        column_name=\"aired\",\n",
    "        new_column_names=[\"start_date\", \"end_date\"],\n",
    "        sep=\",\",\n",
    "    )\n",
    "    .remove_columns(column_names=[\"aired\"])\n",
    "    .str_remove(column_name=\"start_date\", pat=\"'\")\n",
    "    .str_slice(column_name=\"start_date\", start=0, stop=10)\n",
    "    .str_remove(column_name=\"end_date\", pat=\"'\")\n",
    "    .str_slice(column_name=\"end_date\", start=0, stop=11)\n",
    "    .to_datetime(\"start_date\", format=\"%Y-%m-%d\", errors=\"coerce\")\n",
    "    .to_datetime(\"end_date\", format=\"%Y-%m-%d\", errors=\"coerce\")\n",
    "    .fill_empty(columns=[\"rank\", \"popularity\"], value=0)\n",
    "    .filter_on(\"rank != 0 & popularity != 0\")\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>animeID</th>\n",
       "      <th>name</th>\n",
       "      <th>title_english</th>\n",
       "      <th>title_japanese</th>\n",
       "      <th>title_synonyms</th>\n",
       "      <th>type</th>\n",
       "      <th>source</th>\n",
       "      <th>episodes</th>\n",
       "      <th>status</th>\n",
       "      <th>airing</th>\n",
       "      <th>...</th>\n",
       "      <th>synopsis</th>\n",
       "      <th>background</th>\n",
       "      <th>premiered</th>\n",
       "      <th>broadcast</th>\n",
       "      <th>related</th>\n",
       "      <th>producers</th>\n",
       "      <th>genre</th>\n",
       "      <th>studio</th>\n",
       "      <th>start_date</th>\n",
       "      <th>end_date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Cowboy Bebop</td>\n",
       "      <td>Cowboy Bebop</td>\n",
       "      <td>カウボーイビバップ</td>\n",
       "      <td>[]</td>\n",
       "      <td>TV</td>\n",
       "      <td>Original</td>\n",
       "      <td>26.0</td>\n",
       "      <td>Finished Airing</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>In the year 2071, humanity has colonized sever...</td>\n",
       "      <td>When Cowboy Bebop first aired in spring of 199...</td>\n",
       "      <td>Spring 1998</td>\n",
       "      <td>Saturdays at 01:00 (JST)</td>\n",
       "      <td>{'Adaptation': [{'mal_id': 173, 'type': 'manga...</td>\n",
       "      <td>Bandai Visual</td>\n",
       "      <td>Action</td>\n",
       "      <td>Sunrise</td>\n",
       "      <td>1998-04-03</td>\n",
       "      <td>1999-04-24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Cowboy Bebop</td>\n",
       "      <td>Cowboy Bebop</td>\n",
       "      <td>カウボーイビバップ</td>\n",
       "      <td>[]</td>\n",
       "      <td>TV</td>\n",
       "      <td>Original</td>\n",
       "      <td>26.0</td>\n",
       "      <td>Finished Airing</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>In the year 2071, humanity has colonized sever...</td>\n",
       "      <td>When Cowboy Bebop first aired in spring of 199...</td>\n",
       "      <td>Spring 1998</td>\n",
       "      <td>Saturdays at 01:00 (JST)</td>\n",
       "      <td>{'Adaptation': [{'mal_id': 173, 'type': 'manga...</td>\n",
       "      <td>Bandai Visual</td>\n",
       "      <td>Adventure</td>\n",
       "      <td>Sunrise</td>\n",
       "      <td>1998-04-03</td>\n",
       "      <td>1999-04-24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Cowboy Bebop</td>\n",
       "      <td>Cowboy Bebop</td>\n",
       "      <td>カウボーイビバップ</td>\n",
       "      <td>[]</td>\n",
       "      <td>TV</td>\n",
       "      <td>Original</td>\n",
       "      <td>26.0</td>\n",
       "      <td>Finished Airing</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>In the year 2071, humanity has colonized sever...</td>\n",
       "      <td>When Cowboy Bebop first aired in spring of 199...</td>\n",
       "      <td>Spring 1998</td>\n",
       "      <td>Saturdays at 01:00 (JST)</td>\n",
       "      <td>{'Adaptation': [{'mal_id': 173, 'type': 'manga...</td>\n",
       "      <td>Bandai Visual</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>Sunrise</td>\n",
       "      <td>1998-04-03</td>\n",
       "      <td>1999-04-24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Cowboy Bebop</td>\n",
       "      <td>Cowboy Bebop</td>\n",
       "      <td>カウボーイビバップ</td>\n",
       "      <td>[]</td>\n",
       "      <td>TV</td>\n",
       "      <td>Original</td>\n",
       "      <td>26.0</td>\n",
       "      <td>Finished Airing</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>In the year 2071, humanity has colonized sever...</td>\n",
       "      <td>When Cowboy Bebop first aired in spring of 199...</td>\n",
       "      <td>Spring 1998</td>\n",
       "      <td>Saturdays at 01:00 (JST)</td>\n",
       "      <td>{'Adaptation': [{'mal_id': 173, 'type': 'manga...</td>\n",
       "      <td>Bandai Visual</td>\n",
       "      <td>Drama</td>\n",
       "      <td>Sunrise</td>\n",
       "      <td>1998-04-03</td>\n",
       "      <td>1999-04-24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>Cowboy Bebop</td>\n",
       "      <td>Cowboy Bebop</td>\n",
       "      <td>カウボーイビバップ</td>\n",
       "      <td>[]</td>\n",
       "      <td>TV</td>\n",
       "      <td>Original</td>\n",
       "      <td>26.0</td>\n",
       "      <td>Finished Airing</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>In the year 2071, humanity has colonized sever...</td>\n",
       "      <td>When Cowboy Bebop first aired in spring of 199...</td>\n",
       "      <td>Spring 1998</td>\n",
       "      <td>Saturdays at 01:00 (JST)</td>\n",
       "      <td>{'Adaptation': [{'mal_id': 173, 'type': 'manga...</td>\n",
       "      <td>Bandai Visual</td>\n",
       "      <td>Sci-Fi</td>\n",
       "      <td>Sunrise</td>\n",
       "      <td>1998-04-03</td>\n",
       "      <td>1999-04-24</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   animeID          name title_english title_japanese title_synonyms type  \\\n",
       "0        1  Cowboy Bebop  Cowboy Bebop      カウボーイビバップ             []   TV   \n",
       "1        1  Cowboy Bebop  Cowboy Bebop      カウボーイビバップ             []   TV   \n",
       "2        1  Cowboy Bebop  Cowboy Bebop      カウボーイビバップ             []   TV   \n",
       "3        1  Cowboy Bebop  Cowboy Bebop      カウボーイビバップ             []   TV   \n",
       "4        1  Cowboy Bebop  Cowboy Bebop      カウボーイビバップ             []   TV   \n",
       "\n",
       "     source  episodes           status airing  ...  \\\n",
       "0  Original      26.0  Finished Airing  False  ...   \n",
       "1  Original      26.0  Finished Airing  False  ...   \n",
       "2  Original      26.0  Finished Airing  False  ...   \n",
       "3  Original      26.0  Finished Airing  False  ...   \n",
       "4  Original      26.0  Finished Airing  False  ...   \n",
       "\n",
       "                                            synopsis  \\\n",
       "0  In the year 2071, humanity has colonized sever...   \n",
       "1  In the year 2071, humanity has colonized sever...   \n",
       "2  In the year 2071, humanity has colonized sever...   \n",
       "3  In the year 2071, humanity has colonized sever...   \n",
       "4  In the year 2071, humanity has colonized sever...   \n",
       "\n",
       "                                          background    premiered  \\\n",
       "0  When Cowboy Bebop first aired in spring of 199...  Spring 1998   \n",
       "1  When Cowboy Bebop first aired in spring of 199...  Spring 1998   \n",
       "2  When Cowboy Bebop first aired in spring of 199...  Spring 1998   \n",
       "3  When Cowboy Bebop first aired in spring of 199...  Spring 1998   \n",
       "4  When Cowboy Bebop first aired in spring of 199...  Spring 1998   \n",
       "\n",
       "                  broadcast  \\\n",
       "0  Saturdays at 01:00 (JST)   \n",
       "1  Saturdays at 01:00 (JST)   \n",
       "2  Saturdays at 01:00 (JST)   \n",
       "3  Saturdays at 01:00 (JST)   \n",
       "4  Saturdays at 01:00 (JST)   \n",
       "\n",
       "                                             related      producers  \\\n",
       "0  {'Adaptation': [{'mal_id': 173, 'type': 'manga...  Bandai Visual   \n",
       "1  {'Adaptation': [{'mal_id': 173, 'type': 'manga...  Bandai Visual   \n",
       "2  {'Adaptation': [{'mal_id': 173, 'type': 'manga...  Bandai Visual   \n",
       "3  {'Adaptation': [{'mal_id': 173, 'type': 'manga...  Bandai Visual   \n",
       "4  {'Adaptation': [{'mal_id': 173, 'type': 'manga...  Bandai Visual   \n",
       "\n",
       "       genre   studio start_date   end_date  \n",
       "0     Action  Sunrise 1998-04-03 1999-04-24  \n",
       "1  Adventure  Sunrise 1998-04-03 1999-04-24  \n",
       "2     Comedy  Sunrise 1998-04-03 1999-04-24  \n",
       "3      Drama  Sunrise 1998-04-03 1999-04-24  \n",
       "4     Sci-Fi  Sunrise 1998-04-03 1999-04-24  \n",
       "\n",
       "[5 rows x 28 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multi-Step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Data Dictionary\n",
    ">\n",
    ">Heads up the dataset is about 97 mb - if you want to free up some space, drop the synopsis and background, they are long strings, or broadcast, premiered, related as they are redundant or less useful.\n",
    ">\n",
    ">|variable       |class     |description |\n",
    "|:--------------|:---------|:-----------|\n",
    "|animeID        |double    | Anime ID (as in https://myanimelist.net/anime/animeID)          |\n",
    "|name           |character |anime title - extracted from the site.           |\n",
    "|title_english  |character | title in English (sometimes is different, sometimes is missing)          |\n",
    "|title_japanese |character | title in Japanese (if Anime is Chinese or Korean, the title, if available, in the respective language)          |\n",
    "|title_synonyms |character | other variants of the title         |\n",
    "|type           |character | anime type (e.g. TV, Movie, OVA)          |\n",
    "|source         |character | source of anime (i.e original, manga, game, music, visual novel etc.)         |\n",
    "|producers      |character | producers          |\n",
    "|genre          |character | genre         |\n",
    "|studio         |character | studio           |\n",
    "|episodes       |double    | number of episodes           |\n",
    "|status         |character | Aired or not aired      |\n",
    "|airing         |logical   | True/False is still airing          |\n",
    "|start_date     |double    | Start date (ymd)        |\n",
    "|end_date       |double    | End date (ymd)        |\n",
    "|duration       |character | Per episode duration or entire duration, text string        |\n",
    "|rating         |character | Age rating         |\n",
    "|score          |double    | Score (higher = better)       |\n",
    "|scored_by      |double    | Number of users that scored          |\n",
    "|rank           |double    | Rank - weight according to MyAnimeList formula          |\n",
    "|popularity     |double    |  based on how many members/users have the respective anime in their list          |\n",
    "|members        |double    | number members that added this anime in their list         |\n",
    "|favorites      |double    | number members that favorites these in their list          |\n",
    "|synopsis       |character | long string with anime synopsis          |\n",
    "|background     |character | long string with production background and other things          |\n",
    "|premiered      |character | anime premiered on season/year          |\n",
    "|broadcast      |character | when is (regularly) broadcasted         |\n",
    "|related        |character | dictionary: related animes, series, games etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 0: Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = \"https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2019/2019-04-23/raw_anime.csv\"\n",
    "df = pd.read_csv(filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>animeID</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>name</th>\n",
       "      <td>Cowboy Bebop</td>\n",
       "      <td>Cowboy Bebop: Tengoku no Tobira</td>\n",
       "      <td>Trigun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>title_english</th>\n",
       "      <td>Cowboy Bebop</td>\n",
       "      <td>Cowboy Bebop: The Movie</td>\n",
       "      <td>Trigun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>title_japanese</th>\n",
       "      <td>カウボーイビバップ</td>\n",
       "      <td>カウボーイビバップ 天国の扉</td>\n",
       "      <td>トライガン</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>title_synonyms</th>\n",
       "      <td>[]</td>\n",
       "      <td>[\"Cowboy Bebop: Knockin' on Heaven's Door\"]</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>type</th>\n",
       "      <td>TV</td>\n",
       "      <td>Movie</td>\n",
       "      <td>TV</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>source</th>\n",
       "      <td>Original</td>\n",
       "      <td>Original</td>\n",
       "      <td>Manga</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>producers</th>\n",
       "      <td>['Bandai Visual']</td>\n",
       "      <td>['Sunrise', 'Bandai Visual']</td>\n",
       "      <td>['Victor Entertainment']</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>genre</th>\n",
       "      <td>['Action', 'Adventure', 'Comedy', 'Drama', 'Sc...</td>\n",
       "      <td>['Action', 'Drama', 'Mystery', 'Sci-Fi', 'Space']</td>\n",
       "      <td>['Action', 'Sci-Fi', 'Adventure', 'Comedy', 'D...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>studio</th>\n",
       "      <td>['Sunrise']</td>\n",
       "      <td>['Bones']</td>\n",
       "      <td>['Madhouse']</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>episodes</th>\n",
       "      <td>26</td>\n",
       "      <td>1</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>status</th>\n",
       "      <td>Finished Airing</td>\n",
       "      <td>Finished Airing</td>\n",
       "      <td>Finished Airing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>airing</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aired</th>\n",
       "      <td>{'from': '1998-04-03T00:00:00+00:00', 'to': '1...</td>\n",
       "      <td>{'from': '2001-09-01T00:00:00+00:00', 'to': No...</td>\n",
       "      <td>{'from': '1998-04-01T00:00:00+00:00', 'to': '1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>duration</th>\n",
       "      <td>24 min per ep</td>\n",
       "      <td>1 hr 55 min</td>\n",
       "      <td>24 min per ep</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rating</th>\n",
       "      <td>R - 17+ (violence &amp; profanity)</td>\n",
       "      <td>R - 17+ (violence &amp; profanity)</td>\n",
       "      <td>PG-13 - Teens 13 or older</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>score</th>\n",
       "      <td>8.81</td>\n",
       "      <td>8.41</td>\n",
       "      <td>8.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>scored_by</th>\n",
       "      <td>405664</td>\n",
       "      <td>120243</td>\n",
       "      <td>212537</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rank</th>\n",
       "      <td>26</td>\n",
       "      <td>164</td>\n",
       "      <td>255</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>popularity</th>\n",
       "      <td>39</td>\n",
       "      <td>449</td>\n",
       "      <td>146</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>members</th>\n",
       "      <td>795733</td>\n",
       "      <td>197791</td>\n",
       "      <td>408548</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>favorites</th>\n",
       "      <td>43460</td>\n",
       "      <td>776</td>\n",
       "      <td>10432</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>synopsis</th>\n",
       "      <td>In the year 2071, humanity has colonized sever...</td>\n",
       "      <td>Another day, another bounty—such is the life o...</td>\n",
       "      <td>Vash the Stampede is the man with a $$60,000,0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>background</th>\n",
       "      <td>When Cowboy Bebop first aired in spring of 199...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>The Japanese release by Victor Entertainment h...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>premiered</th>\n",
       "      <td>Spring 1998</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Spring 1998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>broadcast</th>\n",
       "      <td>Saturdays at 01:00 (JST)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Thursdays at 01:15 (JST)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>related</th>\n",
       "      <td>{'Adaptation': [{'mal_id': 173, 'type': 'manga...</td>\n",
       "      <td>{'Parent story': [{'mal_id': 1, 'type': 'anime...</td>\n",
       "      <td>{'Adaptation': [{'mal_id': 703, 'type': 'manga...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                0  \\\n",
       "animeID                                                         1   \n",
       "name                                                 Cowboy Bebop   \n",
       "title_english                                        Cowboy Bebop   \n",
       "title_japanese                                          カウボーイビバップ   \n",
       "title_synonyms                                                 []   \n",
       "type                                                           TV   \n",
       "source                                                   Original   \n",
       "producers                                       ['Bandai Visual']   \n",
       "genre           ['Action', 'Adventure', 'Comedy', 'Drama', 'Sc...   \n",
       "studio                                                ['Sunrise']   \n",
       "episodes                                                       26   \n",
       "status                                            Finished Airing   \n",
       "airing                                                      False   \n",
       "aired           {'from': '1998-04-03T00:00:00+00:00', 'to': '1...   \n",
       "duration                                            24 min per ep   \n",
       "rating                             R - 17+ (violence & profanity)   \n",
       "score                                                        8.81   \n",
       "scored_by                                                  405664   \n",
       "rank                                                           26   \n",
       "popularity                                                     39   \n",
       "members                                                    795733   \n",
       "favorites                                                   43460   \n",
       "synopsis        In the year 2071, humanity has colonized sever...   \n",
       "background      When Cowboy Bebop first aired in spring of 199...   \n",
       "premiered                                             Spring 1998   \n",
       "broadcast                                Saturdays at 01:00 (JST)   \n",
       "related         {'Adaptation': [{'mal_id': 173, 'type': 'manga...   \n",
       "\n",
       "                                                                1  \\\n",
       "animeID                                                         5   \n",
       "name                              Cowboy Bebop: Tengoku no Tobira   \n",
       "title_english                             Cowboy Bebop: The Movie   \n",
       "title_japanese                                     カウボーイビバップ 天国の扉   \n",
       "title_synonyms        [\"Cowboy Bebop: Knockin' on Heaven's Door\"]   \n",
       "type                                                        Movie   \n",
       "source                                                   Original   \n",
       "producers                            ['Sunrise', 'Bandai Visual']   \n",
       "genre           ['Action', 'Drama', 'Mystery', 'Sci-Fi', 'Space']   \n",
       "studio                                                  ['Bones']   \n",
       "episodes                                                        1   \n",
       "status                                            Finished Airing   \n",
       "airing                                                      False   \n",
       "aired           {'from': '2001-09-01T00:00:00+00:00', 'to': No...   \n",
       "duration                                              1 hr 55 min   \n",
       "rating                             R - 17+ (violence & profanity)   \n",
       "score                                                        8.41   \n",
       "scored_by                                                  120243   \n",
       "rank                                                          164   \n",
       "popularity                                                    449   \n",
       "members                                                    197791   \n",
       "favorites                                                     776   \n",
       "synopsis        Another day, another bounty—such is the life o...   \n",
       "background                                                    NaN   \n",
       "premiered                                                     NaN   \n",
       "broadcast                                                     NaN   \n",
       "related         {'Parent story': [{'mal_id': 1, 'type': 'anime...   \n",
       "\n",
       "                                                                2  \n",
       "animeID                                                         6  \n",
       "name                                                       Trigun  \n",
       "title_english                                              Trigun  \n",
       "title_japanese                                              トライガン  \n",
       "title_synonyms                                                 []  \n",
       "type                                                           TV  \n",
       "source                                                      Manga  \n",
       "producers                                ['Victor Entertainment']  \n",
       "genre           ['Action', 'Sci-Fi', 'Adventure', 'Comedy', 'D...  \n",
       "studio                                               ['Madhouse']  \n",
       "episodes                                                       26  \n",
       "status                                            Finished Airing  \n",
       "airing                                                      False  \n",
       "aired           {'from': '1998-04-01T00:00:00+00:00', 'to': '1...  \n",
       "duration                                            24 min per ep  \n",
       "rating                                  PG-13 - Teens 13 or older  \n",
       "score                                                         8.3  \n",
       "scored_by                                                  212537  \n",
       "rank                                                          255  \n",
       "popularity                                                    146  \n",
       "members                                                    408548  \n",
       "favorites                                                   10432  \n",
       "synopsis        Vash the Stampede is the man with a $$60,000,0...  \n",
       "background      The Japanese release by Victor Entertainment h...  \n",
       "premiered                                             Spring 1998  \n",
       "broadcast                                Thursdays at 01:15 (JST)  \n",
       "related         {'Adaptation': [{'mal_id': 703, 'type': 'manga...  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(3).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Clean `producers` column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step tries to clean up the `producers` column by removing some brackets ('[]') and trim off some empty spaces\n",
    "\n",
    ">```\n",
    ">clean_df <- raw_df %>% \n",
    ">  # Producers\n",
    ">  mutate(producers = str_remove(producers, \"\\\\[\"),\n",
    "         producers = str_remove(producers, \"\\\\]\"))\n",
    ">```\n",
    "\n",
    "What is mutate? This [link](https://pandas.pydata.org/pandas-docs/stable/getting_started/comparison/comparison_with_r.html) compares R's `mutate` to be similar to pandas' `df.assign`.\n",
    "However, `df.assign` returns a new DataFrame whereas `mutate` adds a new variable while preserving the previous ones.\n",
    "Therefore, for this example, I will compare `mutate` to be similar to `df['col'] = X`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, this is looks like a list of items but in string form"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0               ['Bandai Visual']\n",
       "1    ['Sunrise', 'Bandai Visual']\n",
       "2        ['Victor Entertainment']\n",
       "3               ['Bandai Visual']\n",
       "4          ['TV Tokyo', 'Dentsu']\n",
       "Name: producers, dtype: object"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's see what we trying to remove\n",
    "df.loc[df[\"producers\"].str.contains(\"\\[\", na=False), \"producers\"].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use pandas flavor to create a custom method for just removing some strings so we don't have to use str.replace so many times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "@pf.register_dataframe_method\n",
    "def str_remove(df, column_name: str, pat: str, *args, **kwargs):  # noqa: F811\n",
    "    \"\"\"\n",
    "    Wrapper around df.str.replace\n",
    "    The function will loop through regex patterns\n",
    "    and remove them from the desired column.\n",
    "\n",
    "    :param df: A pandas DataFrame.\n",
    "    :param column_name: A `str` indicating which column\n",
    "        the string removal action is to be made.\n",
    "    :param pat: A regex pattern to match and remove.\n",
    "    \"\"\"\n",
    "\n",
    "    if not isinstance(pat, str):\n",
    "        raise TypeError(\n",
    "            f\"Pattern should be a valid regex pattern. \"\n",
    "            f\"Received pattern: {pat} with dtype: {type(pat)}\"\n",
    "        )\n",
    "    df[column_name] = df[column_name].str.replace(pat, \"\", *args, **kwargs)\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_df = df.str_remove(column_name=\"producers\", pat=\"\\[|\\]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With brackets removed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0               'Bandai Visual'\n",
       "1    'Sunrise', 'Bandai Visual'\n",
       "2        'Victor Entertainment'\n",
       "3               'Bandai Visual'\n",
       "4          'TV Tokyo', 'Dentsu'\n",
       "Name: producers, dtype: object"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_df[\"producers\"].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Brackets are removed. Now the next part\n",
    ">```\n",
    ">  separate_rows(producers, sep = \",\") %>% \n",
    ">```\n",
    "\n",
    "It seems like separate rows will go through each value of the column, and if the value is a list, will create a new row for each value in the list with the remaining column values being the same. This is commonly known as an `explode` method but it is not yet implemented in pandas. We will need a function for this (code adopted from [here](https://qiita.com/rikima/items/c10e27d8b7495af4c159))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "@pf.register_dataframe_method\n",
    "def explode(df: pd.DataFrame, column_name: str, sep: str):  # noqa: F811\n",
    "    \"\"\"\n",
    "    For rows with a list of values,\n",
    "    this function will create new rows\n",
    "    for each value in the list\n",
    "\n",
    "    :param df: A pandas DataFrame.\n",
    "    :param column_name: A `str` indicating which column\n",
    "        the string removal action is to be made.\n",
    "    :param sep: The delimiter.\n",
    "        Example delimiters include `|`, `, `, `,` etc.\n",
    "    \"\"\"\n",
    "\n",
    "    df[\"id\"] = df.index\n",
    "    wdf = (\n",
    "        pd.DataFrame(df[column_name].str.split(sep).fillna(\"\").tolist())\n",
    "        .stack()\n",
    "        .reset_index()\n",
    "    )\n",
    "    # exploded_column = column_name\n",
    "    wdf.columns = [\"id\", \"depth\", column_name]  # plural form to singular form\n",
    "    # wdf[column_name] = wdf[column_name].apply(lambda x: x.strip())  # trim\n",
    "    wdf.drop(\"depth\", axis=1, inplace=True)\n",
    "\n",
    "    return pd.merge(df, wdf, on=\"id\", suffixes=(\"_drop\", \"\")).drop(\n",
    "        columns=[\"id\", column_name + \"_drop\"]\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_df = clean_df.explode(column_name=\"producers\", sep=\",\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now every producer is its own row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0           'Bandai Visual'\n",
       "1                 'Sunrise'\n",
       "2           'Bandai Visual'\n",
       "3    'Victor Entertainment'\n",
       "4           'Bandai Visual'\n",
       "Name: producers, dtype: object"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_df[\"producers\"].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now remove single quotes and a bit of trimming\n",
    ">```\n",
    "  mutate(producers = str_remove(producers, \"\\\\'\"),\n",
    "         producers = str_remove(producers, \"\\\\'\"),\n",
    "         producers = str_trim(producers)) %>% \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_df = clean_df.str_remove(column_name=\"producers\", pat=\"'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll make another custom function for trimming whitespace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "@pf.register_dataframe_method\n",
    "def str_trim(df, column_name: str, *args, **kwargs):  # noqa: F811\n",
    "    \"\"\"Remove trailing and leading characters, in a given column\"\"\"\n",
    "    df[column_name] = df[column_name].str.strip(*args, **kwargs)\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_df = clean_df.str_trim(\"producers\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, here is our cleaned `producers` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0           Bandai Visual\n",
       "1                 Sunrise\n",
       "2           Bandai Visual\n",
       "3    Victor Entertainment\n",
       "4           Bandai Visual\n",
       "Name: producers, dtype: object"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_df[\"producers\"].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Clean `genre` and `studio` Columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do the same process for columns `Genre` and `Studio`\n",
    "\n",
    ">```\n",
    ">  # Genre\n",
    "  mutate(genre = str_remove(genre, \"\\\\[\"),\n",
    "         genre = str_remove(genre, \"\\\\]\")) %>% \n",
    "  separate_rows(genre, sep = \",\") %>% \n",
    "  mutate(genre = str_remove(genre, \"\\\\'\"),\n",
    "         genre = str_remove(genre, \"\\\\'\"),\n",
    "         genre = str_trim(genre)) %>% \n",
    ">  # Studio\n",
    "  mutate(studio = str_remove(studio, \"\\\\[\"),\n",
    "         studio = str_remove(studio, \"\\\\]\")) %>% \n",
    "  separate_rows(studio, sep = \",\") %>% \n",
    "  mutate(studio = str_remove(studio, \"\\\\'\"),\n",
    "         studio = str_remove(studio, \"\\\\'\"),\n",
    "         studio = str_trim(studio)) %>% \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_df = (\n",
    "    clean_df\n",
    "    # Perform operation for genre.\n",
    "    .str_remove(column_name=\"genre\", pat=\"\\[|\\]\")\n",
    "    .explode(column_name=\"genre\", sep=\",\")\n",
    "    .str_remove(column_name=\"genre\", pat=\"'\")\n",
    "    .str_trim(column_name=\"genre\")\n",
    "    # Now do it for studio\n",
    "    .str_remove(column_name=\"studio\", pat=\"\\[|\\]\")\n",
    "    .explode(column_name=\"studio\", sep=\",\")\n",
    "    .str_remove(column_name=\"studio\", pat=\"'\")\n",
    "    .str_trim(column_name=\"studio\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Resulting cleaned columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>genre</th>\n",
       "      <th>studio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Action</td>\n",
       "      <td>Sunrise</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Adventure</td>\n",
       "      <td>Sunrise</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Comedy</td>\n",
       "      <td>Sunrise</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Drama</td>\n",
       "      <td>Sunrise</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sci-Fi</td>\n",
       "      <td>Sunrise</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       genre   studio\n",
       "0     Action  Sunrise\n",
       "1  Adventure  Sunrise\n",
       "2     Comedy  Sunrise\n",
       "3      Drama  Sunrise\n",
       "4     Sci-Fi  Sunrise"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_df[[\"genre\", \"studio\"]].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Clean `aired` column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `aired` column has something a little different. In addition to the usual removing some strings and whitespace trimming, we want to separate the values into two separate columns `start_date` and `end_date`\n",
    "\n",
    ">```r\n",
    ">  # Aired\n",
    "  mutate(aired = str_remove(aired, \"\\\\{\"),\n",
    "         aired = str_remove(aired, \"\\\\}\"),\n",
    "         aired = str_remove(aired, \"'from': \"),\n",
    "         aired = str_remove(aired, \"'to': \"),\n",
    "         aired = word(aired, start = 1, 2, sep = \",\")) %>% \n",
    "  separate(aired, into = c(\"start_date\", \"end_date\"), sep = \",\") %>% \n",
    "  mutate(start_date = str_remove_all(start_date, \"'\"),\n",
    "         start_date = str_sub(start_date, 1, 10),\n",
    "         end_date = str_remove_all(start_date, \"'\"),\n",
    "         end_date = str_sub(end_date, 1, 10)) %>%\n",
    "  mutate(start_date = lubridate::ymd(start_date),\n",
    "         end_date = lubridate::ymd(end_date)) %>%\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will create some custom wrapper functions to emulate R's `word` and use pyjanitor's `deconcatenate_column`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    {'from': '1998-04-03T00:00:00+00:00', 'to': '1...\n",
       "1    {'from': '1998-04-03T00:00:00+00:00', 'to': '1...\n",
       "2    {'from': '1998-04-03T00:00:00+00:00', 'to': '1...\n",
       "3    {'from': '1998-04-03T00:00:00+00:00', 'to': '1...\n",
       "4    {'from': '1998-04-03T00:00:00+00:00', 'to': '1...\n",
       "Name: aired, dtype: object"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Currently looks like this\n",
    "clean_df[\"aired\"].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "@pf.register_dataframe_method\n",
    "def str_word(  # noqa: F811\n",
    "    df,\n",
    "    column_name: str,\n",
    "    start: int = None,\n",
    "    stop: int = None,\n",
    "    pat: str = \" \",\n",
    "    *args,\n",
    "    **kwargs,\n",
    "):  # noqa: F811\n",
    "    \"\"\"\n",
    "    Wrapper around `df.str.split`,\n",
    "    with additional `start` and `end` arguments\n",
    "    to select a slice of the list of words.\n",
    "\n",
    "    :param df: A pandas DataFrame.\n",
    "    :param column_name: A `str` indicating which column\n",
    "        the split action is to be made.\n",
    "    :param start: optional An `int` for the start index of the slice\n",
    "    :param stop: optional  An `int` for the end index of the slice\n",
    "    :param pat: String or regular expression to split on.\n",
    "        If not specified, split on whitespace.\n",
    "\n",
    "    \"\"\"\n",
    "    df[column_name] = df[column_name].str.split(pat).str[start:stop]\n",
    "    return df\n",
    "\n",
    "\n",
    "@pf.register_dataframe_method\n",
    "def str_join(df, column_name: str, sep: str, *args, **kwargs):  # noqa: F811\n",
    "    \"\"\"\n",
    "    Wrapper around `df.str.join`\n",
    "    Joins items in a list.\n",
    "\n",
    "    :param df: A pandas DataFrame.\n",
    "    :param column_name: A `str` indicating which column\n",
    "        the split action is to be made.\n",
    "    :param sep: The delimiter. Example delimiters\n",
    "        include `|`, `, `, `,` etc.\n",
    "    \"\"\"\n",
    "    df[column_name] = df[column_name].str.join(sep)\n",
    "    return df\n",
    "\n",
    "\n",
    "@pf.register_dataframe_method\n",
    "def str_slice(  # noqa: F811\n",
    "    df, column_name: str, start: int = None, stop: int = None, *args, **kwargs\n",
    "):  # noqa: F811\n",
    "    \"\"\"\n",
    "    Wrapper around `df.str.slice\n",
    "    Slices strings.\n",
    "\n",
    "    :param df: A pandas DataFrame.\n",
    "    :param column_name: A `str` indicating which column\n",
    "        the split action is to be made.\n",
    "    :param start: 'int' indicating start of slice.\n",
    "    :param stop: 'int' indicating stop of slice.\n",
    "    \"\"\"\n",
    "    df[column_name] = df[column_name].str[start:stop]\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_df = (\n",
    "    clean_df.str_remove(column_name=\"aired\", pat=\"\\{|\\}|'from':\\s*|'to':\\s*\")\n",
    "    .str_word(column_name=\"aired\", start=0, stop=2, pat=\",\")\n",
    "    .str_join(column_name=\"aired\", sep=\",\")\n",
    "    # .add_columns({'start_date': clean_df['aired'][0]})\n",
    "    .deconcatenate_column(\n",
    "        column_name=\"aired\", new_column_names=[\"start_date\", \"end_date\"], sep=\",\"\n",
    "    )\n",
    "    .remove_columns(column_names=[\"aired\"])\n",
    "    .str_remove(column_name=\"start_date\", pat=\"'\")\n",
    "    .str_slice(column_name=\"start_date\", start=0, stop=10)\n",
    "    .str_remove(column_name=\"end_date\", pat=\"'\")\n",
    "    .str_slice(column_name=\"end_date\", start=0, stop=11)\n",
    "    .to_datetime(\"start_date\", format=\"%Y-%m-%d\", errors=\"coerce\")\n",
    "    .to_datetime(\"end_date\", format=\"%Y-%m-%d\", errors=\"coerce\")\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>start_date</th>\n",
       "      <th>end_date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1998-04-03</td>\n",
       "      <td>1999-04-24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1998-04-03</td>\n",
       "      <td>1999-04-24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1998-04-03</td>\n",
       "      <td>1999-04-24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1998-04-03</td>\n",
       "      <td>1999-04-24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1998-04-03</td>\n",
       "      <td>1999-04-24</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  start_date   end_date\n",
       "0 1998-04-03 1999-04-24\n",
       "1 1998-04-03 1999-04-24\n",
       "2 1998-04-03 1999-04-24\n",
       "3 1998-04-03 1999-04-24\n",
       "4 1998-04-03 1999-04-24"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Resulting 'start_date' and 'end_date' columns with 'aired' column removed\n",
    "clean_df[[\"start_date\", \"end_date\"]].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Filter out unranked and unpopular series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's drop the unranked or unpopular series with pyjanitor's `filter_on`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First fill any NA values with 0 and then filter != 0\n",
    "clean_df = clean_df.fill_empty(column_names=[\"rank\", \"popularity\"], value=0).filter_on(\n",
    "    \"rank != 0 & popularity != 0\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### End Result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>animeID</th>\n",
       "      <th>name</th>\n",
       "      <th>title_english</th>\n",
       "      <th>title_japanese</th>\n",
       "      <th>title_synonyms</th>\n",
       "      <th>type</th>\n",
       "      <th>source</th>\n",
       "      <th>episodes</th>\n",
       "      <th>status</th>\n",
       "      <th>airing</th>\n",
       "      <th>...</th>\n",
       "      <th>synopsis</th>\n",
       "      <th>background</th>\n",
       "      <th>premiered</th>\n",
       "      <th>broadcast</th>\n",
       "      <th>related</th>\n",
       "      <th>producers</th>\n",
       "      <th>genre</th>\n",
       "      <th>studio</th>\n",
       "      <th>start_date</th>\n",
       "      <th>end_date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Cowboy Bebop</td>\n",
       "      <td>Cowboy Bebop</td>\n",
       "      <td>カウボーイビバップ</td>\n",
       "      <td>[]</td>\n",
       "      <td>TV</td>\n",
       "      <td>Original</td>\n",
       "      <td>26.0</td>\n",
       "      <td>Finished Airing</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>In the year 2071, humanity has colonized sever...</td>\n",
       "      <td>When Cowboy Bebop first aired in spring of 199...</td>\n",
       "      <td>Spring 1998</td>\n",
       "      <td>Saturdays at 01:00 (JST)</td>\n",
       "      <td>{'Adaptation': [{'mal_id': 173, 'type': 'manga...</td>\n",
       "      <td>Bandai Visual</td>\n",
       "      <td>Action</td>\n",
       "      <td>Sunrise</td>\n",
       "      <td>1998-04-03</td>\n",
       "      <td>1999-04-24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Cowboy Bebop</td>\n",
       "      <td>Cowboy Bebop</td>\n",
       "      <td>カウボーイビバップ</td>\n",
       "      <td>[]</td>\n",
       "      <td>TV</td>\n",
       "      <td>Original</td>\n",
       "      <td>26.0</td>\n",
       "      <td>Finished Airing</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>In the year 2071, humanity has colonized sever...</td>\n",
       "      <td>When Cowboy Bebop first aired in spring of 199...</td>\n",
       "      <td>Spring 1998</td>\n",
       "      <td>Saturdays at 01:00 (JST)</td>\n",
       "      <td>{'Adaptation': [{'mal_id': 173, 'type': 'manga...</td>\n",
       "      <td>Bandai Visual</td>\n",
       "      <td>Adventure</td>\n",
       "      <td>Sunrise</td>\n",
       "      <td>1998-04-03</td>\n",
       "      <td>1999-04-24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Cowboy Bebop</td>\n",
       "      <td>Cowboy Bebop</td>\n",
       "      <td>カウボーイビバップ</td>\n",
       "      <td>[]</td>\n",
       "      <td>TV</td>\n",
       "      <td>Original</td>\n",
       "      <td>26.0</td>\n",
       "      <td>Finished Airing</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>In the year 2071, humanity has colonized sever...</td>\n",
       "      <td>When Cowboy Bebop first aired in spring of 199...</td>\n",
       "      <td>Spring 1998</td>\n",
       "      <td>Saturdays at 01:00 (JST)</td>\n",
       "      <td>{'Adaptation': [{'mal_id': 173, 'type': 'manga...</td>\n",
       "      <td>Bandai Visual</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>Sunrise</td>\n",
       "      <td>1998-04-03</td>\n",
       "      <td>1999-04-24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Cowboy Bebop</td>\n",
       "      <td>Cowboy Bebop</td>\n",
       "      <td>カウボーイビバップ</td>\n",
       "      <td>[]</td>\n",
       "      <td>TV</td>\n",
       "      <td>Original</td>\n",
       "      <td>26.0</td>\n",
       "      <td>Finished Airing</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>In the year 2071, humanity has colonized sever...</td>\n",
       "      <td>When Cowboy Bebop first aired in spring of 199...</td>\n",
       "      <td>Spring 1998</td>\n",
       "      <td>Saturdays at 01:00 (JST)</td>\n",
       "      <td>{'Adaptation': [{'mal_id': 173, 'type': 'manga...</td>\n",
       "      <td>Bandai Visual</td>\n",
       "      <td>Drama</td>\n",
       "      <td>Sunrise</td>\n",
       "      <td>1998-04-03</td>\n",
       "      <td>1999-04-24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>Cowboy Bebop</td>\n",
       "      <td>Cowboy Bebop</td>\n",
       "      <td>カウボーイビバップ</td>\n",
       "      <td>[]</td>\n",
       "      <td>TV</td>\n",
       "      <td>Original</td>\n",
       "      <td>26.0</td>\n",
       "      <td>Finished Airing</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>In the year 2071, humanity has colonized sever...</td>\n",
       "      <td>When Cowboy Bebop first aired in spring of 199...</td>\n",
       "      <td>Spring 1998</td>\n",
       "      <td>Saturdays at 01:00 (JST)</td>\n",
       "      <td>{'Adaptation': [{'mal_id': 173, 'type': 'manga...</td>\n",
       "      <td>Bandai Visual</td>\n",
       "      <td>Sci-Fi</td>\n",
       "      <td>Sunrise</td>\n",
       "      <td>1998-04-03</td>\n",
       "      <td>1999-04-24</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   animeID          name title_english title_japanese title_synonyms type  \\\n",
       "0        1  Cowboy Bebop  Cowboy Bebop      カウボーイビバップ             []   TV   \n",
       "1        1  Cowboy Bebop  Cowboy Bebop      カウボーイビバップ             []   TV   \n",
       "2        1  Cowboy Bebop  Cowboy Bebop      カウボーイビバップ             []   TV   \n",
       "3        1  Cowboy Bebop  Cowboy Bebop      カウボーイビバップ             []   TV   \n",
       "4        1  Cowboy Bebop  Cowboy Bebop      カウボーイビバップ             []   TV   \n",
       "\n",
       "     source  episodes           status airing  ...  \\\n",
       "0  Original      26.0  Finished Airing  False  ...   \n",
       "1  Original      26.0  Finished Airing  False  ...   \n",
       "2  Original      26.0  Finished Airing  False  ...   \n",
       "3  Original      26.0  Finished Airing  False  ...   \n",
       "4  Original      26.0  Finished Airing  False  ...   \n",
       "\n",
       "                                            synopsis  \\\n",
       "0  In the year 2071, humanity has colonized sever...   \n",
       "1  In the year 2071, humanity has colonized sever...   \n",
       "2  In the year 2071, humanity has colonized sever...   \n",
       "3  In the year 2071, humanity has colonized sever...   \n",
       "4  In the year 2071, humanity has colonized sever...   \n",
       "\n",
       "                                          background    premiered  \\\n",
       "0  When Cowboy Bebop first aired in spring of 199...  Spring 1998   \n",
       "1  When Cowboy Bebop first aired in spring of 199...  Spring 1998   \n",
       "2  When Cowboy Bebop first aired in spring of 199...  Spring 1998   \n",
       "3  When Cowboy Bebop first aired in spring of 199...  Spring 1998   \n",
       "4  When Cowboy Bebop first aired in spring of 199...  Spring 1998   \n",
       "\n",
       "                  broadcast  \\\n",
       "0  Saturdays at 01:00 (JST)   \n",
       "1  Saturdays at 01:00 (JST)   \n",
       "2  Saturdays at 01:00 (JST)   \n",
       "3  Saturdays at 01:00 (JST)   \n",
       "4  Saturdays at 01:00 (JST)   \n",
       "\n",
       "                                             related      producers  \\\n",
       "0  {'Adaptation': [{'mal_id': 173, 'type': 'manga...  Bandai Visual   \n",
       "1  {'Adaptation': [{'mal_id': 173, 'type': 'manga...  Bandai Visual   \n",
       "2  {'Adaptation': [{'mal_id': 173, 'type': 'manga...  Bandai Visual   \n",
       "3  {'Adaptation': [{'mal_id': 173, 'type': 'manga...  Bandai Visual   \n",
       "4  {'Adaptation': [{'mal_id': 173, 'type': 'manga...  Bandai Visual   \n",
       "\n",
       "       genre   studio start_date   end_date  \n",
       "0     Action  Sunrise 1998-04-03 1999-04-24  \n",
       "1  Adventure  Sunrise 1998-04-03 1999-04-24  \n",
       "2     Comedy  Sunrise 1998-04-03 1999-04-24  \n",
       "3      Drama  Sunrise 1998-04-03 1999-04-24  \n",
       "4     Sci-Fi  Sunrise 1998-04-03 1999-04-24  \n",
       "\n",
       "[5 rows x 28 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_df.head()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
